Papers with parameter generator
Contextual Parameter Generation for Universal Neural Machine Translation (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to multilingual neural machine translation lack language-specific parameterization. |
| Approach: | They propose a modification to existing neural machine translation models that allows for language specific parameterization and domain adaptation. |
| Outcome: | The proposed model surpasses state-of-the-art for both the IWSLT-15 and IWSTL-17 datasets and can perform zero-shot translation. |
Weight Distillation: Transferring the Knowledge in Neural Network Parameters (2021.acl-long)
Copied to clipboard
| Challenge: | Knowledge distillation is an effective method for model acceleration and compression. |
| Approach: | They propose to use parameters to distill knowledge from large neural networks to small ones . they propose to do this by using a parameter generator to transfer the knowledge to a small neural network . |
| Outcome: | The proposed method learns a small network 1.88 2.94x faster than the large network but with competitive BLEU points. |
Privacy-Preserving Reasoning with Knowledge-Distilled Parametric Retrieval Augmented Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing RAG systems require uploading local documents to the cloud, resulting in inference latency and poor generalization on out-of-distribution (OOD) inputs. |
| Approach: | They propose a generalizable knowledge-distilled parametric RAG model aligned with standard RAG in document structure and parameter activation. |
| Outcome: | The proposed model outperforms baselines in accuracy and generalizes well on out-of-distribution (OOD) data. |